Performance and qualii 



Over the last three years, concerns about the performance and 
quality of higher education have risen to the top of the policy agenda. 
During the 1980s, these concepts were the subject of both scholarly 
research (see for example: Lawrence and Green, 1980; Lindsay, 1982; 
Astin, 1985)andpolicy-orientedstudy(forexample:Linkeetal, 1984 
and Bourke, 1986), but it was only in the latter part of the decade that 
performance and quality started to receive sustained attention from 
policy makers. 

One significant step in this process was the commissioning in 1989, 
of the Performance Indicators Research Group to: 
develop and trial a broad range of quantitative indicators suitable 
for evaluating relative performance in higher education at both 
system and institutional levels and to report on their practicability, 
data requirements and appropriate conditions of use. (Linke, 1991: 
xi). 

The Research Group built on earlier work in identifying potential 
indicators, in emphasising compatibility with existing data sources, 
and in focussing on collective rather than individual performance. 
The Report (Linke, 1991) advocated the use of performance indica¬ 
tors as an aid in determining how well institutions or departments 
have achieved their goals. The Report examined a range of indicators 
which were not claimed to be comprehensive but which were intended 
to provide a starting point formonitoring institutional performance in 
teaching and research within a context of relevant background factors. 
The Report argued that indicators could provide measures of effi¬ 
ciency and effectiveness but that they could not serve as the sole 
source of information on quality. 

Concern with quality was given momentum by the Report of the 
Senate Standing Committee on Employment, Education and Training 
(1990). This Report posed questions about the quality of the education 
that students experience and the quality of graduates in terms of their 
specific professional competence and more generalised “higher level 
abilities”. This Report differed from many of the earlier discussions 
by not focussing on precise definitions and measurement, but being 
concerned instead with the nature and value of the skills acquired. 

The concern with quality intensified with an Invitational Workshop 
in 1991 (Anwyl, 1992), a Ministerial statement on Quality and 
Diversity (Baldwin, 1991) and most recently the publication of a set 
of Discussion Papers on The Quality of Higher Education by the 
Higher Education Council (1992a) followed by the Council’s Draft 
and Final Advice papers (Higher Education Council, 1992b, 1992c). 

This paper explores some of the linkages between the concepts of 
performance and quality, the concerns that are currently being ex¬ 
pressed about them, and the dangers that must be avoided if they are 
to be usefully applied in improving higher education. 



The notion of organisational “performance” is a rather elusive one. 
In everyday usage “performance” relates to a level of achievement in 
some activity. In relation to organisations, assessing “performance” 
involves asking what the organisation is accomplishing and how well 
it is carrying out its tasks. In other words, organisational performance 
embodies the notions of effectiveness and efficiency. 

Delving into technical definitions of effectiveness and efficiency is 


a complex process (for further discussion, see Lindsay, 1983a), and 
it is sufficient to note that the main focus of attention is on an 
organisation’s activities and outcomes, and how resources are used to 
support them. Provided that the organisation concerned conforms to 
certain requirements, obtaining measures of effectiveness and effi¬ 
ciency can be quite straightforward. 

There are three specific requirements to be met: 

• the organisation must have inputs and outputs that can be identi¬ 
fied, measured and priced, 

• decision-makers must understand and be able to control the 
organisation’s processes, 

• decision-makers must be seeking to maximise outputs and min¬ 
imise costs. 
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When these conditions apply, effectiveness may be regarded as a 
measure of the extent to which goals have been achieved, and 
efficiency, as a measure of the extent to which outputs have been 
achieved at least cost. While such definitions are standard in manage¬ 
ment theory, many types of organisations, including universities, do 
not fully conform to the three underlying requirements. 

In higher education, multiple goals in relation to excellence, equity, 
access, and diversity, are pursued simultaneously. These goals are 
intrinsically ambiguous and contentious, and the outcomes produced 
are not readily susceptible to measurement or pricing. Nor do those 
who make decisions about goals, processes and resource use, have a 
good understanding of the “production” relationships and firm con¬ 
trol over them (for further discussion see: Cohen and March, 1974; 

Lindsay, 1983b; de Weert, 1988). 

These features underlie the difficulties of using performance indi¬ 
cators in higher education. A consideration of the Report of the 
Performance Indicators Group (Linke, 1991) can be used to illustrate 
the difficulties. 

The Report recommends 23 indicators classified in three groups: 

I. Indicators of institutional Context (10) 

Academic and general staff provision 
Cl Equivalent full time academic staff 

C3 Equivalent lull time student load 
C4 Student/staff ratio 

Student demand and background characteristics 
C5 Student preference ratio 

C7 Student acceptance ratio 

C8 Average student entry score 

Financial resource distribution 
C9 Derivation of recurrent income 

CTO Distribution of recurrent expenditure 
Cl 1 Academic activity cost per student 
Cl 2 Total recurrent cost per student 

II. Performance Indicators (II) 

Teaching and learning 

PI Perceived teaching quality 

P2 Student progress rate 

P3 Program completion rate 

P5 Research higher degree productivity rate 


P6 Graduate employment status 

Research and professional services 

P7a Number of research grants 

P7b Staff supported from external research grants 

P8 Value of research grants 

P9 Average publication rate 

P10 Productivity rate of other original works 

P11 Paid consultancy rate 

III. Participation and Social Equity Indicators (2) 

51 Academic staff gender ratio 

52 Commencing student gender ratio 

The Research Group saw its role as primarily technical and it made 
its major contribution in identifying weaknesses and inconsistencies 
in current measures and data sets, and in developing more adequate 
definitions and consistent and reliable data requirements. The Report 
claimed that the indicators proposed offer “a reasonable coverage of 
all tbe major aspects of teaching and research which are currently 
amenable to routine quantitative analysis” (Linke, 1991: xiv). The 
problem is that many important aspects are not amenable to measure¬ 
ment. 

The ten Context Indicators represent the “background factors” 
against which performance in teaching and research are to be meas¬ 
ured. The staff and financial indicators are clearly input measures, and 
while providing some useful improvements in the data available to * 
decision-makers, they do not directly illuminate performance. The 
indicators of Student Demand are more difficult to classify. In one 
sense they are “inputs” to the teaching process, but in another sense 
they could be regarded as performance indicators since they reflect to 
some extent an institution’s reputation and its performance in attract¬ 
ing good students. 

In the Report eleven measures are designated as “performance”' 
indicators. Of the five teaching indicators, three are improved ver¬ 
sions of the standard notions of progress and completion rates. 
Another indicator gives initial employment status information, also in 
a fairly conventional form. The only “new” measure is an indicator of 
“Perceived Teaching Quality” which uses student questionnaire data 
on a valuable but limited range of teaching facets. However, the 
measure only provides a single perspective on the quality of teaching. 
The information relates solely to student perceptions of their experi¬ 
ence. It is not, for example, complemented by data on adequacy of 
course content and organisation, or most importantly, student out¬ 
comes. As this indicator is the single measure in this area, there is a 
real danger of neglecting other important questions, such as whether 
the course content is up-to-date, accurate or relevant to the intended 
purpose, and whether the course produces graduates with the neces¬ 
sary professional competencies. 

The research indicators are quite conventional. They involve the 
numbers and value of giants received, publication rates and the 
number of consultancies. Such measures are quite limited in their 
representation of the outcomes of research, but have the virtue of 
being easy to measure. Peer review, a well-established and effective 
method o f assessing research performance, does not readily lend itself 
to simple quantitative measures. Thus, measures of the significance 
or impact of research have largely eluded researchers. 

The Research Group also made little progress with the Participation 
and Equity Indicators which merely cover basic gender data. Simple 
and suitable definitions of some of the other categories of disadvan¬ 
tage, such as disability and socio-economic status, have not been easy 
to develop. 

Quality in higher education 

With the recent placing of “quality” alongside “performance” as a 
central concern, a further complex concept has been added to the 
debate. The term “quality” is used in many different ways. Often a 
particular approach is adopted in order to promote a particular 
political or educational position (Cave et al, 1988; de Weert, 1988; 


Westerheijden, 1990). Some discussions of quality focus mainly on 
performance; that is, on identifying inputs, processes and outputs, and 
on obtaining quantitative measures of them. Others focus more on 
how to make sound judgements about those processes and outcomes 
including those which are largely intangible. 

The everyday notion of assessing “quality” involves making a 
judgement of “worth”. This process entails the application of external 
standards, either implicit or explicit, about what constitutes “poor”, 
“good” or “excellent”. While simple in concept, many works have 
been written on the notions of quality and the judging of excellence 
in higher education (for example: Astin, 1985; Lawrence and Green, 
1980; Lindsay and Neumann, 1988). 

The performance view of quality emphasises the measurement of 
simply-defined outputs and tends to neglect the role that judgement 
plays. This type of view has been prominent in Government thinking 
in recent years, and has found expression in the studies commissioned 
from Linke et al (1984), Bourke (1986) and the Commonwealth 
Tertiary Education Commission (1986). 

In its first response to the most recent Government reference on 
quality, the Higher Education Council (HEC) (1991) gave most 
emphasis to quality as residing in outcomes, with outcomes being 
defined as the attributes graduates should have acquired. Higher 
education is expected to provide the community with trained gradu¬ 
ates who have specific task skills, more general professional skills and 
generic higher-level skills such as knowing how to learn, solve 
problems, and think logically. This rather narrow outcomes view and 
the associated concern for definition and measurement, links the HEC 
approach to quality directly to the concepts which underpin the Linke 
Report on performance indicators. While outcomes provide one 
useful line of approach in assessing quality, there are alternative and 
complementary concepts including the “reputational”, “value-added” 
and “content” approaches (Lindsay, 1992). 

However, the Higher Education Council also identified several 
other notions of quality including the view that it is recognisable but 
not easily defined, and that its assessment is dependent on compara¬ 
tive judgements and varying values and perspectives. 

In identifying these factors, the Council has acknowledged the 
extent to which different parties will make different assessments of 
quality as a result of their different perspectives and priorities, and the 
consequent emphasis that should be placed on the judgements that 
stakeholders make about the quality of higher education. In recogni¬ 
tion of this variety, the Council invited and published a set of 
discussion papers prepared by the major stakeholders in higher 
education: the National Union of Students (NUS), the Federated 
Australian University Staff Association and Union of Australian 
College Academics (FAUSA/UACA), the Council of Australian 
Postgraduate Associations (CAPA) and the Australian Vice-Chancel¬ 
lors ’ Committee (AVCC). 

These discussion papers (Higher Education Council, 1992a) all 
display a broader view of quality than the HEC itself, with each 
acknowledging the inevitable diversity in judgements about quality 
that arises from the different perspectives on goals and values of the 
various stakeholders. The student groups, NUS and CAPA, do not 
support a simple outcomes approach but advocate a focus on improv¬ 
ing educational processes by attention to the many factors that 
influence student learning. 

The staff unions do indicate support for an outcomes approach 
provided multiple and inter-related outcomes are accommodated and 
the approach is sensitive to contextual factors. On the other hand, the 
AVCC is largely concerned with the impact that declining resource 
levels have had on quality, with defending the existing quality 
management processes. 

Approaches that treat quality as residing in outcomes and perform¬ 
ance have the same conceptual and measurement problems as identi¬ 
fied earlier in relation to performance indicators. The underlying 
model in each case is a systems or “production” model of inputs, 
processes and outputs. Despite its obvious utility and widespread use, 
a systems model of education is not really a simple one. The inputs and 
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outputs are abstractions not real entities and many alternative and 
sometimes competing conceptualisations are possible. For example, 
staff inputs may be represented by numbers, as in student-staff ratios, 
or by salaries, as in cost per student ratios. For some purposes, 
measures of staff qualifications and skills, or contact hours maybe the 
appropriate ones. The different measures may yield different results 
in an analysis, and identifying how the measures actually relate to the 
underlying conceptualisation of the educational process may be 
important in selecting among the alternatives. 

Despite the extensive range of views canvassed in its consultations, 
the Higher Education Council’s (1992c) final advice to the Minister 
still lacks a clear focus and a comprehensive and coherent position. 
The general thrust of the HEC’s original Discussion Paper has been 
refined and elaborated, but the various parts do not interlock to 
provide a consistent and systematic blueprint for a national quality 
assurance system. 



In the dominant views of performance and quality, the elements of 
the “production” process employed are those that can be most easily 
quantified. Hence, popular indicators include completion rates, stu¬ 
dent-staff ratios, and publications per staff member. By de-emphasis¬ 
ing the exercise of judgement about what these indicators mean in 
educational terms, they are commonly represented as measures of 
quality. 

However, while the recent advances in defining and implementing 
indicators do provide a better coverage of what is measurable, they are 
not sufficient to constitute a new and useful tool for higher education 
decision making. Quantitative indicators are simply not able to 
capture adequately the many dimensions of performance and quality. 

In complex multi-dimensional systems, quantitative indicators are 
at best partial measures of institutional and system performance and 
quality. They focus attention on measurable and hence limited 
elements at the expense of those that are intangible but also arguably 
more significant. Unfortunately, the precise definition and quantifi¬ 
cation of indicators often gives them a quite unjustified importance in 
comparison with more global and judgemental assessments which 
may in fact capture performance and quality more adequately. 

For example, although the Performance Indicators Report argues 
that its set of indicators is sufficiently comprehensive, diverse and 
reliable to be implemented by institutions, the Report gives little 
consideration to evaluating how well its indicators do in fact capture 
“performance”. 

The Report argues that performance indicators should “assist in 
determining how well a particular institution or department has 
achieved its respective goals” (Linke, 1991: xiii). However, it does 
not examine higher education’s goals or how the proposed indicators 
specifically relate to them. Instead, the Report relies heavily on the 
assumption that higher education’s goals are well known, clearly 
defined, and subject to sufficient agreement among the parties 
involved to be taken for granted in performance assessment. 

There is a partial acknowledgement that this may not be the case in 
the suggestion that different institutions have different ideals and 
different performance expectations, and that therefore judgements of 
performance must take account of these factors instead of merely 
making a direct assessment on the basis of the relevant indicator 
values. 

The Research Group also attempts to distance the notion of “qual¬ 
ity” and acceptable ways of assessing it from quantifiable measures 
of performance. The Report argues that “provided the indicators are 
based on effective quality assurance procedures and attuned to 
important educational aims their impact should nevertheless be a 
constructive one...” (Linke, 1991: 131). Flowever, this approach begs 
questions which are quite central to any performance assessment - 
how quality is assessed and assured, and how well can quantitative 
indicators be attuned to all, or even most, important educational aims. 

Adopting any particular concept of an educational input or output 
inevitably involves an over-simplification. Simple quantitative meas¬ 


ures cannot adequately represent educational performance or quality. 
A wide variety of measures and the overall judgements of interested 
parties are necessary to make adequate assessments. Judgements are 
necessary both in determining the significance of individual measures 
and in reaching global assessments from a diverse array of informa¬ 
tion both quantitative and non-quantitative. 

At the heart of the broad approach advocated here is the belief that 
judgements, including those which involve a global weighing up of 
many factors, some unquantifiable, play an indispensable part in 
assessing educational performance and quality. Allied to this view is 
the concern that a pre-occupation with quantifying performance has 
distorted our conception of educational processes and outcomes at the 
expense of important but intangible dimensions which cannot be 
captured by simple indicators. Hence, performance indicators should 
be regarded as having only a restricted role in quality assessment in 
higher education. 



In seeking to use performance indicators in higher education 
considerable care should be exercised. The available indicators are 
simply not of sufficient substance in relation to higher education’s 
goals, processes and outcomes, to provide a significant improvement 
in our capacity to measure performance or quality. In higher 
education, many important characteristics are, unfortunately, simply 
not measurable. 

While considerable progress has been made in recent years in 
developing more adequate definitions and consistent and reliable data 
requirements (see for example, Cave et al, 1988; Dochy et al, 1990; 
Linke, 1991), there have been no conceptual or empirical advances 
that significantly improve our representations of performance or 
quality. 

The current debates about higher education’s performance and 
quality show an unfortunate tendency to separate technical questions 
about definition and measurement from questions about the educa¬ 
tional basis of what is being attempted, and from the more political 
questions such as how the results are to be used, and whose purposes 
are to be served. 

As a consequence, some of the proposals in the Report on Perform¬ 
ance Indicators and the Higher Education Council’s discussions of, 
and final advice on, quality must be regarded as potentially damaging 
to higher education as they give undue prominence to measurable but 
limited aspects of higher education at the expense of more significant 
but intangible aspects. Their use could thus divert the attention of 
system and institutional decision makers from core concerns to the 
pursuit of good indicator scores. 

If indicators are used with appropriate recognition of their limita¬ 
tions they can provide a useful starting point for further investigation. 
Measures which relate to outcomes, the efficient use of resources and 
where possible, the “value-added” by educational processes, can be of 
considerable use to decision makers. The danger arises from the 
spurious weight which is often given to quantitative measures (as for 
example with student-staff ratios), and their consequent inappropriate 
use in making decisions about allocations to and within institutions. 

To counter this danger institutions should strengthen their institu¬ 
tional research capabilities, and while routinely collecting quantita¬ 
tive measures such as completion rates and research grant levels, they 
should also obtain stakeholder judgements about quality, including 
global assessments from institutional reputation ratings, graduate and 
employer satisfaction surveys, and peer review processes. 

In addition, safeguards should be put in place to reduce the potential 
misuse of evaluative information. These should include clear state¬ 
ments about the limitations of the measures and judgements utilised, 
consultation about standards and procedures, careful selection and 
training of participants, and subjecting the processes and results 
themselves to appraisal. 

In summary, current approaches to performance and quality tend to 
over-simplify higher education’s role and the notion of outcomes; 
over-emphasise measurement at the expense of judgement; and make 


insufficient allowance for diverse and conflicting stakeholder judge¬ 
ments. Only a broad approach to performance and quality can 
accommodate the diversity of goals and values that characterises 
higher education and its stakeholders, and in particular, the conflict¬ 
ing judgements that result. 

The broadening of the national debate on quality that occurred 
through the involvement of the stakeholder groups provides the 
opportunity to advance our thinking on quality beyond a focus on 
performance indicators by giving adequate attention to non-quantifi- 
able information. In participating in the Government’s quality audit, 
the academic community should give close attention to ensuring that 
this opportunity is not lost. It is essential to ensure that measurement 
processes are not overvalued to the neglect of judgement processes, 
and a mechanism is developed for accommodating and making use of 
the widely varying judgements that result from quality assessments 
from different stakeholder perspectives. In preparing for the quality 
audits, institutions should design a practical approach to assessing the 
quality of educational programs that is consistent with the broad 
perspective on performance and quality that has been outlined in this 
paper. One such approach would be to assemble a set of quantitative 
indicators andjudgements on particular program elements made from 
various perspectives, and then submit them for consideration by a 
stakeholder-based committee which can then make both specific and 
overall judgements about program quality in a systematic and coher¬ 
ent manner. 
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